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ABSTRACT 



This paper emphasizes research-based reasons for adding 
audio to multimedia presentations. The first section summarizes suggestions 
from a review of research on the effectiveness of audio media when 
accompanied by other forms of media; types of research studies (e.g., 
evaluation, intra-medium, and aptitude treatment interaction studies) are 
also described. Recommended teaching techniques for three learning modalities 
(kinesthetic, auditory, and visual) are outlined in the second section. The 
next section provides examples of the use of audio in multimedia 
presentations, focusing on the functions of three elements of sound: speech, 
including narration, dialogue, and direct address; sound effects, including 
contextual and narrative functions; and music. Literal and non- literal sounds 
are discussed in the fourth section, followed by a section that addresses the 
functions of sound in radio and television (to provide information, establish 
outer orientation, and establish inner orientation) . The sixth section covers 
four roles that audio media play in multimedia production: picture defines 
sound; sound defines picture; sound parallels picture; and sound 
counterpoints picture. Aesthetic factors of sound- - figure ground, sound 
perspective, and sound continuity- -are described in the next section. Two 
concluding sections list software for recording audio and automating online 
multimedia presentations and provide links to resources on audio file types 
and platform compatibility. (MES) 
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This paper emphasizes research-based reasons for adding audio to multimedia presentations. Media 
examples, links for gaining more in-depth knowledge concerning this topic and procedures for adding 
audio media from any PC (Macintosh or Windows-based) can be referenced from the following URL: 
http:// www . mtsu . edu/ -itres . 

This topic was chosen because the author has observed many faculty members adding still images, video, 
and links to web and non-web documents to their presentations but few faculty members have taken 
advantage of the learning provided by audio media integration. In preparation for designing a class around 
this topic the author has begun the following investigation into research in the effective use of audio media 
in multimedia presentations. 



[Top] 



Why Should Educators Include Audio Media in their Presentations? 



Thompson, Simonson, and Hargrave in Educational Technology, A Review of the Research (AECT) 
reviewed studies conducted on the effectiveness of audio media on learning when accompanied by other 
forms of media. Their review of the studies offered the following suggestions : 
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• Students can learn when various forms of audio media accompany other media. 

• The use of background music can increase achievement for some learners, but is probably not 
necessary. 

• The use of audio media with other media may enhance the understanding of content material. 

• The meaning of a visual message is often ambiguous and subject to personal interpretation. The use 
of words to direct attention is essential. 

• With visuals, some verbalization is better than no visuals, but there is no optimum amount. Slow 
speeds for transmitting verbal information are favored but they can be too slow. Rates need to be 
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tailored to fit the student and their familiarity with the content. 

• When narration is accompanied by video the optimum rate of the narration appears to be slower. 

• The audio channel is much more capable of maintaining attention if it is used as an interjection on 
the visual channel rather than being continuously parallel with the visual. 

Types of Research Studies 

Evaluation Research is usually the first type of research done for each media type. Evaluation Research 
tries to determine whether people can learn from a particular form of media. It was found that given 
favorable conditions, students could learn from any instructional media. Media comparison studies were 
then conducted to try to determine if one media type was more effective in learning than another type 
(1920’s - 1960’s). Media comparison studies produced insignificant results. For every study that showed 
that a new medium was better, another study showed the opposite . 

Intra-medium studies were the next type of research conducted. Intra-medium studies examined the 
interactions among student, task, and specific media characteristics in terms of what happens when these 
variables were manipulated. They compare alternative methods of using a particular medium. According to 
Thompson, Simonson, and Hargrave the design of these studies were based on Saloman’s 

observation that the effectiveness of a medium depends on the nature of the instruction. The major research 
question in these studies was "Which are the most effective instructional approaches using this medium?" 

A particular medium was used in all groups participating in the study. The independent variable was the 
instructional approach, not the medium itself . 

The next type of research conducted was Aptitude Treatment Interaction Studies. Aptitude Treatment 
Interaction Studies attempted to take into account student aptitudes in the research design. Media 
researchers accepted a new paradigm. This paradigm acknowledged the interaction that occurs between 
external stimuli (presented by media) and internal cognitive processes that support learning (Clark, 1988). 
Information about a learner was helpful in adapting instruction in order to provide an environment in 
which particular learners can thrive. Clark tells us that media by themselves do not affect learning but 
rather it is the particular qualities of media or a specific medium that "affects particular cognitive processes 
that are relevant for students with specific aptitudes to learn particular knowledge or skills". Research in 
Aptitude Treatment Interaction Studies have led people to recognize "the importance of different learning 
styles and methods of processing information as well as the correlation that exists between learner 
variables and content treatments. 

Hop] 

Learning Styles 

Gordon (1998) researched the relevance of learner characteristics and learning styles when planning law 
school courses. He identified the four major learning modalities, kinesthetic, tactual, auditory, and visual. 
Kinesthetic learners and auditory learners seemed to benefit the most by the integration of audio media into 
teaching strategies. Gordon’s table below provides a list of suggested teaching techniques based on 
learning modalities. He does not provide data on the tactual learner . 
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Modality 


Teaching technique 


Kinesthetic 


This student needs a combination of stimuli. The manipulation of material 
along with the accompanying sights and sounds (words and numbers seen 
and spoken) will make a big difference to this student. 


Auditory 


This student will benefit from hearing audio tapes, rote oral practice, 
lecture or a class discussion. This student will benefit from tutoring 
another or delivering an explanation to his/her study group or to the 
teacher. 


Visual 


This student will benefit from worksheets, workbooks, and texts. Given 
some time alone with a book, this student may learn more than the class. 



Examples: Effective Use of Audio in Multimedia Presentations 

Lee Daniels (1995), in his paper Audio Vision: Audio-Visual Interaction in Desktop Multimedia lists the 
three elements of sound used in multimedia presentations. He explains how these elements should be used 
to stimulate learning through the functions they serve and roles they provide. Daniels’ ideas are 
summarized in the next few pages. 

Elements of sound 

The three audio elements in multimedia production are speech, sound effects, and music. Silence and its 
effects also need to be considered in planning projects. The various functions of these elements are 
discussed in the next three sections of this paper. It is important that the educator use care in considering 
the audience when using audio media to evoke source images in students’ minds. If the students have not 
been thoroughly exposed to the source that produces the sound the effect and learning will be lost. 

Audio Element: Speech 

Narration, dialogue, and direct address are three functions of the speech element. Their functions will be 
discussed below. 

Narration 



Narrative speech can be used to: 
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Deliver concrete information: Concrete information could include directions for completing a project 
or possibly descriptive information that relates to the image being displayed. When narration is 
presented with text, the text and narration must be exactly the same. Discrepancies may result in 
distraction and cause interference in learning the material which translates to less retention or 
misinterpretation of material. 

Replace text: 

Narration is most useful, as a replacement of text when screen space is limited and the addition of 
text would reduce the visual impact of the page. Narration saves screen space and visual clutter. 
Direct viewer’s attention: 

The image displayed on the screen may need to be the focal point. Narration is used to direct viewer 
attention to the image being displayed rather than forcing the viewer to alternate between viewing an 
image and reading the text explanation. 
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Narration and dialogue together can: 

Affect intensity by setting the pace. Pace effects the intensity of emotion during a presentation. 

Examples include: 

• Fast moving narration adds to the intensity of time lapsed animation. Slow moving narration 
complements the sober mood of a funeral. 

• Fast paced dialogue between two characters can reflect tension, anger, excitement, or nervousness. 
Smooth, even paced dialogue reflects friendliness, relaxation, and confidence. 

Affect the listener’s perception through changes in tone quality. 

Examples include: 

• Bright and present narration is perceived to be closer and more intimate and trustworthy. 

• Speech that sounds dull and distant would have the opposite effect. 

Direct Address 

Direct address refers to the character speaking directly to the audience. 

Examples include: 

• Some TV commercials that talk directly to the audience. 

• Speeches 

• Newscasts 

Audio Element: Sound Effects 

Sound effects can function contextually and narratively. 

Contextual Function 

When sound effects have a contextual function the sound effect interprets the visual as it appears. 
Examples include: 

• a dog barking or a dog begging for a treat 

• the roar of a jet or airplane engine in normal flight or taking off, or the sound of a jet having engine 
trouble 

Narrative Function 

When sound effects have a narrative function the sound effect adds more to the image’s apparent 
information. The functions of the narrative format can be broken down even further into those that provide 
descriptive effects and those that serve a commentary function. 

Narrative Function: Descriptive Effect 
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Sound effects contribute to the subtle aspects of an image. Subtle aspects are those features that are hard to 
define or perceive but that contribute greatly to the emotional effect of the image or scene. 

Examples include: 

• The sounds of gentle ocean surf which may include gulls, people playing, and boat sounds used to 
set a particular mood. 

• The sound of a violent ocean surf and warning sirens sounding in the background. The mood of this 
example would be quite different than the mood of the first example. 

• Imagine a picture of a metal triangle being struck with a metal beater. The audience may hear a clink 
indicating that the triangle may be made of inferior materials or they may hear a clear ringing sound 
indicating that the triangle is made out of quality materials. Perhaps the audience only hears a thud, 
indicating that the triangle only looked like it was made of metal but might have actually been made 
of wood or plastic. 

• Imagine a picture of a cymbal player crashing a pair of cymbals. If a clear ringing sound is heard the 
teacher knows that the player is holding the cymbals correctly as they are being played. If a muffled 
crash is heard followed by little ringing the teacher assumes that the player is not holding the 
cymbals correctly and may be placing too much of his/her hands on the cymbals as they are being 
played. 

Narrative Function: Commentative Sounds 

Commentative sounds also tell more about an image but the information is usually unrelated to the visual 
itself. 



Example: 

• "Imagine a program about air pollution and a scene of city traffic. Treating and blending the car 
engines to ‘sputter’ and ‘cough’ comments on the detrimental effects that air pollution has on the air 
we breathe" . 

Audio Element: Music 

Music is very effective in communicating complicated emotions and moods. Functions of music in 
multimedia presentations include establishing locale or time, identifying characters and events, acting as a 
transition element between contrasting scenes, and setting the mood and pace of presentations. Examples 
of each function are listed below . 

• Locale: Music can define a locale with ethnic melodies. 

• Time: Music can establish time with musical elements that suggest a period in history such as the 
1960’s or the Roman era. 

• Identification: Music can identify characters and events with recurring themes. A short musical 
phrase or specific sound effect can be used to signal the appearance of a person, action, or situation. 
This is sometimes called leitmotiv (German for "leading motive") . 

• Transitions: Music can be used to connect one idea or scene to another. It can also smoothe the transition to a contrasting 
theme. It prepares the audience by letting them know that something is going to change. 

• Pace: Music can be used to establish the pace of the presentation. This pace can parallel the visual 
media or provide counterpoint to signify tension or irony. 
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Silence 

Every moment of a multimedia presentation does not need to be filled with sound. Silence can be used to 
set a mood or to provide a moment for reflection. Producers of multimedia presentations need to consider 
the use of silence as well as the use of the other audio elements when designing a presentation. As stated 
earlier in this paper, the audio channel is much more capable of maintaining attention if it is used as an 
interjection on the visual channel rather than being continuously parallel with the visual. 

Haul 

Literal Sounds and Non-Literal Sounds 

Zettl (1999), in his text Sight Sound Motion. Applied Media Aesthetics breaks down sounds into two 
categories, literal sounds and non-literal sounds. Literal sounds include speech and environmental sounds. 
Non-literal sounds include background music and other sounds that seem to influence feeling in some way 
. All of the sound elements listed in the previous section can be used in a literal or non-literal way. Think 
about how they can be used as you read the definitions below. 

Literal Sounds 

Literal sounds are referential. They convey a specific literal meaning. They refer the listener to the 
sound-producing source. Conversations or the sound of rush-hour traffic refer to people and automobiles. 
Literal sounds can be source-connected or source-disconnected. The audience can see the sound-producing 
source when they hear source-connected sounds. The source of source-disconnected sound is off-screen. 
The listener visualizes the source of the source-disconnected sound. 

Non-Literal Sounds 

Non-literal sounds are not intended to refer to a particular source or convey literal meaning. They are 
deliberately source-disconnected and do not evoke a visual image of the sound-producing source. The 
following sound effects, boings, hisses, and whams, usually used in cartoons, are non-literal sounds. 
Romantic music played in the background during a love scene or the rhythmic musical themes behind 
newscasts are also non-literal sounds. Non-literal sounds are called "nodiegetic" which means they occupy 
non-story space . 

Summary 

Non-literal and literal sounds are often combined in the same scene. Sounds can be literal or non-literal 
depending on their context within the presentation or scene. Music played in the background to influence 
the mood of a scene would be considered non-literal sound. Music played, as the audience watches the 
orchestra musicians perform, would be considered literal sound. 

IlQEl 

Functions of Sound as Used in Radio and Television 

Zettl, (1999) has broken down the use of sound in radio and television production into three categories. He 
tells us that sound functions to provide information, establish outer orientation, and establish inner 
orientation . These functions should also be considered in multimedia production. 
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Zettl established that information is imparted through speech in the form of dialogue, direct address, and 
narration. Direct address refers to the character speaking directly to the audience. We see examples of 
direct address everyday on TV in commercials, speeches, and newscasts. Earlier in this paper Daniels’ 
functions of the audio element, speech, are listed as narration and dialog. Note that Zettl has added the 
function of direct address. The function narration can be source-connected (on camera) or 
source-disconnected (off camera). 

Establish Outer Orientation 

Sound establishes the outer orientation of a scene by orientating the scene in space, time, situation, and 
external event condition. Orientation in space refers to the location, the spatial environment (sounds 
produced in a large room sound different than those produced in a small room) and off-screen space 
(sounds of people and objects not seen on-screen). Morning sounds such as an alarm clock ringing and 
bacon frying distinguishes the time of day. Situation orientated sounds refer to those that describe a 
specific situation. These sounds may be predictive such as those used to indicate the recurring appearance 
of a person or action (leitmotiv), or they may signify the coming of an event such as danger 
(non-recurring). 

Establish Inner Orientation 

Sounds that establish inner orientation set the mood, establish the internal condition, and provide energy 
and structure to the scene. Mood can be set using music or some sort of non-musical, electronically 
produced sound. Sounds that indicate an unstable environment or the feelings of a person establish the 
internal condition. Sounds provide or increase the aesthetic energy of a scene. This energy affects the 
emotions of the viewers. 

Structure is the interaction between the rhythmic structure of the sound versus that of the visual source. 
The interplay between the video and audio structures can be parallel, irregular, or highly independent of 
each other. If the sound and visual structure are considered parallel they move at the same rate. In irregular 
structures the rates of the visual and audio structures may vary. They may move together for a period of 
time and then move separately. If the visual and audio elements move independently of one another they 
provide contrast and interaction among the media types. \ 

IlQEl 

Roles of Audio Media in Multimedia Production 



Daniels lists four roles that audio media play in multimedia production. These roles are: picture defines 
sound , sound defines picture , sound parallels picture , and sound counterpoints picture . Each role is 
described below. Examples are provided . 

Picture defines sound 

When a picture defines a sound the sound is defined when the visual image is so strong that the 
accompanying sound is a literal translation of the image. This is very similar to the descriptive function of 
sound effects described earlier in this paper. 

• Example: 
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• A raging storm, with crashing waves and bent palms demands a soundtrack that consists of wind, 
surf, and rain sound effects. Audio is supportive of the dominant visual, reinforcing the image. 

• A quiet beach with small calm waves demands a soundtrack that consists of soft surf sounds and 
sounds of children playing. 

Sound defines picture 

When the sound defines the picture the sound is so distinctive that the listener forms an image of the 
source in his or her mind before the image is displayed . Examples are listed below. Remember to consider 
the background knowledge of your students when you use sound in this manner. 

• Multimedia program on the Brazilian rain forest 

• A still image of the jungle interior is accompanied by the solitary sounds of the environment: 

• Rainfall 

• Bird calls 

• Other animals 

• Lively ethnic music 

• Distinctive Sounds Produced by Objects 

• Sound of chain saws 

• Sound of various types of machinery 

Sound parallels picture 

Sound parallels picture is the most common relationship between audio and visual elements. The audio 
element combines with the visual element to create a mood or deliver information that is more potent than 
either element alone . 

• Example: The sounds of battle with gunshots, cannon, and anguished screams complement the visual 
of a battle scene. The ferocity and destruction of war is conveyed by both media separately, but is 
intensified by both elements together. 

Sound counterpoints picture 

Sound counterpoints the visual image when both media elements contain unrelated information that creates 
an effect that is not conveyed by either media element alone. Previously Zettl (1999) told us that visual and 
audio elements moving independently of each other provide contrast and interactivity among media types. 

• Example: In a presentation on the civil rights movement, irony is created when a visual montage of 
segregated public facilities is underscored by a reading of the United States Constitution. 
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Aesthetic Factors of Sound 



There are three basic aesthetic factors of sound use in any scene. They are figure ground, sound 
perspective, and sound continuity. Figure ground is the most important sound in the scene, the sound that 
is emphasized. All other sounds will be in the background. Sound perspective refers to close-up sounds 
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matching close up visuals and distant sounds matching distant visuals. Sound continuity is established 
when the intended volume and quality of sound is maintained over a series of events. Sound continuity is 
usually achieved by maintaining the same level of ambient (background) sounds. The figure ground may 
vary in intensity but the background sounds provide the continuity. 

ITopl 

Software 

Mac: Recording Audio PC: Recording Audio Multi-platform: Recording Audio 

• Simple Sound • Sound • QuickTime 

• Sound Edit Recorder: • Real Audio 

• Sound Forge • Narration recorded from within 

PowerPoint 



Software for Automating Online Multimedia Presentations 

• Synchronized Multimedia Integration Language (SMIL): 
http://webreview.com/wr/pub/1999/03/12/feature/index.html 

• RealAudio/Video: http://www.realaudio.com 

• Top Class: http://www.wbtsystems.com/ 

• The Sync-O-Matic 3000: http://www.egr.msu.edu/-crs/projects/syncomat/ 



UqeI 

Audio File Types and Platform Compatibility 

• Editing Visual and Audio Media Class (MTSU): 
http://www.mtsu.edu/~itres/itt/VisualAudioHandout/page7.html 

• Duquesne University’s Digital Duke: http://the-duke.duq-duke.duq.edu/notes/LECTURES/sec2.htm 
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